Add Model Name and System Fingerprint to llm_output in _convert_response_to_chat_result
#84
Conversation
Enhance ChatResult output with model details and token usage
| usage = response.get("usage", {}) | ||
| return ChatResult(generations=generations, llm_output=usage) | ||
| llm_output = { | ||
| "token_usage": response.get("usage", {}), |
There was a problem hiding this comment.
Can we rename this to usage so it doesn't change the API for existing users?
| "token_usage": response.get("usage", {}), | |
| "usage": response.get("usage", {}), |
There was a problem hiding this comment.
I did as @bbqiu suggested, so usage remains intact
There was a problem hiding this comment.
I also created a new unit test, since the existing ones were not altered by this change
There was a problem hiding this comment.
Can we also update tests here: https://github.com/databricks/databricks-ai-bridge/blob/main/integrations/langchain/tests/unit_tests/test_chat_models.py#L84
Thank you for contributing. :)
|
Please reapprove, I fixed the linting error ( P.S. I'm confused about this particular linting error: did you change the linting configuration recently? Otherwise it doesn't make sense to me |
|
Apologies, ruff cli was doing something weird. I created a PR on my forked branch and the linting is now fixed. |
This PR enhances
_convert_response_to_chat_resultby including the model name and system fingerprint inllm_output. The model name is particularly important for observability tools like LangFuse, which rely onllm_output["model_name"]for tracking.Currently, Databricks does not provide the model name in the metadata—only the endpoint name. This change ensures that external models are properly identified, improving observability and logging.
Changes:
model_nametollm_output, defaulting toself.modelif not in the response.system_fingerprinttollm_outputfor additional metadata.llm_outputto includetoken_usageseparately for better structure.Why?
When using external models with Databricks, the returned metadata only includes the endpoint name, not the actual model name. This change allows users to extract the correct model name for logging and monitoring.
Testing:
self.modelifresponse["model"]is missing.